A Multithreaded Runtime System With Thread Migration for Distributed Memory Parallel Computing

Multithreading is very effective at tolerating the latency of remote memory accesses in distributed memory parallel computers, but does nothing to reduce the number or cost of those memory accesses. Compiler techniques and runtime approaches, such as caching remote memory accesses and prefetching, are often used to reduce the number of remote memory accesses. Another approach to reduce the number of remote memory access messages is thread migration, which exploits spatial locality by making the entire memory space of any processing element (PE) local. Nomadic Threads are compiler-generated fine-grain threads that migrate between PEs to access data. The runtime system that implements the Nomadic Threads architecture has several attractive features, including support for migrating significant amounts of work, the ability to mix caching and migratory remote memory accesses, and low scheduling complexity. Our straightforward implementation shows that migration may be easily added to many multithreading runtime systems. We also present some unique behavior characteristics of migratory programs to help determine when migration is appropriate or not.